Skip to content

Make stored day-records timezone-independent with CalendarDay#85

Merged
kyleve merged 10 commits into
mainfrom
cursor/calendarday-timezone-fix
Jul 15, 2026
Merged

Make stored day-records timezone-independent with CalendarDay#85
kyleve merged 10 commits into
mainfrom
cursor/calendarday-timezone-fix

Conversation

@kyleve

@kyleve kyleve commented Jul 15, 2026

Copy link
Copy Markdown
Owner

Why

Manually entered overrides and missing-day backfills (and dismissed issues) were persisted keyed by an absolute Date instant meant to be midnight in the writer's time zone. After a time-zone change (e.g. New York → San Francisco), startOfDay for a given logical date resolves to a different instant, so stored keys stop matching recomputed day keys and previously-resolved data errors reappear.

A DB export confirmed it: of 160 backfilled manual days, 9 were logged in New York (Eastern-midnight instants) and now resolve to the previous calendar day in Pacific — exactly the reappearing gaps.

What

Introduce CalendarDay (a timezone-independent YYYY-MM-DD value) as the identity of a logical day, and key every stored user record and day comparison on it. Date is kept only where an instant is genuinely needed (GPS sample→day bucketing, calendar-grid geometry, display), always derived via startOfDay(in:).

  • CalendarDayComparable/Hashable/Codable (encodes as the ISO string), Date conversions, pure-Gregorian arithmetic/ranges, and a legacy start-of-day recovery init.
  • Threaded through the domainDayPresence.day, RegionDayLocations.day, YearReport, MissingDays/MissingDayRange, PresenceCalendar, the three detectors, and DataIssueID.storageKey (now prefix:YYYY-MM-DD, so dismissals survive travel). EvidenceReader day keys too.
  • PersistenceSDManualDay.dayKey (ISO) is the canonical key; dateKey is retained as informational history + toValue()'s recovery source. Store CRUD keys manual days by CalendarDay (lexicographic ISO range for year scoping). WhereStore/DayJournal/ReportReader/WidgetDataReader updated to match.
  • WhereUI / WhereIntents consumers project CalendarDay → Date only for display and DatePicker bindings.
  • Backup format v2 — manualDays changed shape (dateday); a pre-v2 build now refuses a v2 archive with the graceful "newer version" message. The current reader still imports v1 (recovering the legacy date).

Existing data (no boot-time migration)

Correctness for pre-CalendarDay rows comes from read recovery, not a migration: SDManualDay.toValue() recovers a legacy dayKey-less row's CalendarDay from its dateKey in UTC (timezone-stable), so the reappearing-gaps bug is fixed on read alone. Persisting canonical dayKeys is a one-time, manual backup export → replace-import (the export recovers correct days; import rewrites rows with dayKey).

An earlier revision of this PR added a boot-time StoreMigration framework + CalendarDayMigration; it was removed as over-built for pre-release (single developer's data, ongoing boot-against-CloudKit risk) and unnecessary for correctness. The general, durable successor — per-entity schema versioning with lazy upcasting + minReaderVersion + read-repair — is captured as a follow-up in Where/TODOs.md.

Known gaps (acceptable pre-release, noted in WhereCore/AGENTS.md + TODO):

  • A legacy dayKey-less row is invisible to the dayKey-range queries until re-imported (allManualDays()/export still see it via recovery), so the year report shows those days after the one-time reimport.
  • Legacy dismissal keys (epoch, not ISO) aren't recovered on read, so a pre-CalendarDay dismissal can reappear until re-dismissed or fixed in the export round-trip. (The reported export has 0 dismissals.)
  • GPS-derived days still bucket by the current calendar at read time by design, so a dismissed GPS-only border-drift/abrupt-change issue can still shift with travel; only user-asserted records are pinned.

Testing

  • New: CalendarDayTests, TimeZoneIndependenceTests, plus a BackupServiceTests case proving a v1 date-keyed manifest imports to the correct CalendarDay.
  • ./swiftformat --lint clean; the full Stuff-iOS-Tests scheme passes on iPhone 17 / iOS 26.2 (the multi-bundle host case).

Notes

  • Read recovery uses UTC, which is correct for continental writers; the extreme ±12h-offset edges are the only case it could be off by a day (irrelevant for the US data here).

kyleve added 8 commits July 14, 2026 12:01
Introduce a timezone-independent calendar day (year/month/day) as the
stable identity of a logical day, with Date conversions, ISO string
round-tripping, pure-Gregorian arithmetic/ranges, and a legacy
start-of-day recovery init for the upcoming data migration.

Plan step: Add CalendarDay + tests.
Make the timezone-independent CalendarDay the identity of a logical day
throughout the Where domain, so user-asserted records and dismissals stop
drifting onto a different day (and reappearing as data errors) when the
device changes time zones.

- DayPresence is now keyed by `day: CalendarDay` (with `startOfDay(in:)`
  and a `date:in:` producer convenience); Codable decodes legacy `date`
  manifests via UTC recovery.
- Aggregation, YearReport, RegionDayLocations, MissingDays/MissingDayRange,
  PresenceCalendar, and the data-issue detectors/IDs/keys all work in
  CalendarDay. DataIssueID.storageKey is now `prefix:YYYY-MM-DD`, stable
  across travel. EvidenceReader day keys are CalendarDay.
- Persistence: SDManualDay gains canonical `dayKey` (ISO string); `dateKey`
  is retained as informational + the migration source. Store CRUD keys
  manual days by CalendarDay (lexicographic ISO range for year scoping);
  WhereStore/DayJournal/ReportReader/WidgetDataReader updated to match.
- WhereUI/WhereIntents consumers project CalendarDay -> Date only for
  display and DatePicker bindings.

Adds a reusable boot-time migration framework (StoreMigration registry +
App-Group version marker + SwiftDataStore.runPendingMigrations, hooked in
WhereBootstrap.makeServices) and the v1 CalendarDayMigration: backfills
dayKey from legacy instants via a +12h noon-nudge recovery (robust to the
writer's zone) and rewrites legacy epoch dismissal keys to ISO form.

Plan steps: DayPresence/aggregation; MissingDays + calendar; detection keys;
persistence; migration framework; CalendarDayMigration; WhereUI consumers.
- WhereCore/AGENTS.md: invariants for CalendarDay (logical-day identity vs
  Date-for-instants) and the boot-time StoreMigration pattern.
- Where/AGENTS.md: CalendarDay note in Dates & presentation.
- WhereCore/README.md: CalendarDay + StoreMigration/CalendarDayMigration in
  the API tour; dismissals key on a TZ-independent storageKey.
- Add TimeZoneIndependenceTests: a backfilled day read in a different time
  zone stays present and is not re-flagged as missing; dismissal keys are
  timezone-independent.

Plan step: Docs (AGENTS invariant + READMEs) + TZ-independence regression test.
Fixes review finding #1: an SDManualDay synced from a still-legacy build
has no dayKey, so the dayKey-range predicate dropped it before toValue()'s
recovery could run — the day vanished from the year report (yet still
appeared in a backup) and survived clearYear.

manualDays(in:), clear(in:manualDays:), and clearManualDay(_:) now fold in
dayKey == nil rows via a shared keylessManualDays helper (recovered with
toValue(), filtered in Swift), so such a synced-in day stays visible and
deletable. Normally empty; a stopgap for the mixed-version sync window.

Adds SwiftDataStore tests for read/clear-year/clear-day recovery, and a
Where/TODOs.md entry for the durable successor (per-entity schema versioning
with lazy upcasting + minReaderVersion + read-repair).
Fixes review finding #2. manualDays are now keyed by `day` (CalendarDay)
instead of a `date` instant, changing the manifest shape, so the format
version must bump — otherwise a pre-CalendarDay build reads a new archive
as v1 and hits an opaque DecodingError on the missing `date` key instead
of the graceful `unsupportedFormatVersion` message.

The current reader still imports v1 archives (DayPresence decodes the
legacy `date` and recovers its calendar day). Adds a BackupServiceTests
case proving a v1 date-keyed manifest imports to the correct CalendarDay
(also covers the legacy-import half of finding #3).
Fixes review finding #4. insertLegacyManualDay is purely a test fixture
(it writes a deliberately invalid, dayKey-less row), so per the repo
convention it belongs in #if DEBUG + @_spi(Testing) and must not ship in
release. runMigrations stays ungated — it's the impl runPendingMigrations
calls, not test-only.
Addresses review finding #5 (informational). Clarifies in WhereCore/AGENTS.md
and DataIssueID.storageKey that CalendarDay pins *stored user records* and
their keys across a time-zone change, but GPS samples are still bucketed by
the current calendar at read time — so a GPS-only border-drift/abrupt-change
dismissal can still shift with its re-bucketed day. Only user-asserted
records are travel-proof; bucketing GPS by a fixed zone is intentionally not
done.
Fixes review finding #6. The parser validated field widths and 1...12 /
1...31 ranges but accepted well-formed-but-impossible dates like
2026-02-31. It now round-trips the components through the Gregorian
calendar and rejects any that normalize (Feb 31, Feb 29 in a non-leap
year, Apr 31), while real leap days still parse. Guards against a corrupt
persisted key decoding as a bogus day.
@kyleve kyleve marked this pull request as ready for review July 15, 2026 18:40
Reconsidered for pre-release: the boot-time StoreMigration framework +
CalendarDayMigration was over-built for a single developer's data and
carried ongoing boot-against-CloudKit risk. It isn't needed for
correctness — SDManualDay.toValue() already recovers a legacy dayKey-less
row's CalendarDay from dateKey (in UTC, so timezone-stable), which fixes
the reappearing-gaps bug on read alone. Persisting canonical dayKeys is
handled by a one-time backup export -> replace-import (the export recovers
correct days; import rewrites rows with dayKey).

Removes:
- StoreMigration / CalendarDayMigration and their tests
- SwiftDataStore.runPendingMigrations / runMigrations / insertLegacyManualDay
  + the App-Group version marker, and the WhereBootstrap boot hook
- the keyless-row query recovery (manualDays(in:)/clear/clearManualDay) and
  its tests — a dayKey-less row is invisible to range queries until the
  one-time reimport rewrites it (allManualDays/export still sees it)

Keeps: CalendarDay + threading, toValue() recovery, DataIssueID ISO keys,
BackupArchive v2. Docs (AGENTS/README/TODOs) updated to describe the
export/reimport approach, note that legacy dismissal keys aren't recovered
on read, and keep the per-entity-versioning system as the durable successor.
/// decides the day boundaries).
public init(from date: Date, in calendar: Calendar) {
let parts = calendar.dateComponents([.year, .month, .day], from: date)
self.init(year: parts.year ?? 1, month: parts.month ?? 1, day: parts.day ?? 1)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this throw or crash or return nil? Returning 1 for any of these is silent data corruption

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posted by an AI agent on kyleve's behalf.

Good catch — changed to a preconditionFailure when the calendar returns no year/month/day (an impossible state for a Gregorian calendar + a concrete Date) rather than silently defaulting to Jan 1. Fixed in 2a84f3a.

(lhs.year, lhs.month, lhs.day) < (rhs.year, rhs.month, rhs.day)
}

public init(from decoder: any Decoder) throws {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need custom encoders and decoders? Can't the usual standard compiler provided item do this for us?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eg, there's no reason to even round trip to ISO. Just persist the three int fields. If we've initialized, it's a valid day.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posted by an AI agent on kyleve's behalf.

Removed — Codable is now compiler-synthesized over the three int fields (no custom init(from:)/encode(to:)). Fixed in 2a84f3a.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posted by an AI agent on kyleve's behalf.

Done — dropped the ISO round-trip on decode; synthesized Codable persists the three ints directly and trusts validity-by-construction. (init?(iso:) and description stay for the SDManualDay.dayKey / DataIssueID.storageKey string form.) One side effect: the backup manifest's day becomes an object instead of a string — fine since there's no released v2 data. Fixed in 2a84f3a.

let interval = aggregator.yearInterval(year: year)
try await store.perform { try await store.clear(in: interval) }
let dayRange = CalendarDay(year: year, month: 1, day: 1)
... CalendarDay(year: year, month: 12, day: 31)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of hardcoding this can we derive it from the calendar?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posted by an AI agent on kyleve's behalf.

Added CalendarDay.lastDay(ofYear:) / yearRange(_:) and reused them here and in ReportReader, WidgetDataReader, MissingDaysDetector, and MissingDays instead of hardcoding Dec 31 in each spot. Since CalendarDay is Gregorian-only, Dec 31 is invariant, so the helper encodes that once rather than reintroducing a live Calendar dependency for a constant. Fixed in 2a84f3a.

- CalendarDay(from:in:): trap via preconditionFailure when the calendar
  returns no year/month/day (an impossible state) instead of silently
  defaulting to Jan 1.
- Drop the custom Codable / ISO round-trip; a CalendarDay is valid by
  construction, so use compiler-synthesized Codable over the three ints.
  (Persistence/keys still use `description`/`init?(iso:)`; only the backup
  manifest's `day` shape changes, string -> object — no released v2 data.)
- Add CalendarDay.lastDay(ofYear:) / yearRange(_:) helpers and use them in
  DayJournal.clearYear, ReportReader, WidgetDataReader, MissingDaysDetector,
  and MissingDays instead of hardcoding Dec 31 in each spot.
@kyleve kyleve enabled auto-merge (squash) July 15, 2026 19:36
@kyleve kyleve merged commit 7bbaafc into main Jul 15, 2026
2 checks passed
kyleve added a commit that referenced this pull request Jul 15, 2026
## What

Trims the largest `AGENTS.md` files down to invariants, hard-fought lessons, and instructions — dropping structure enumerations agents re-derive from the manifests in their first minute of searching — and regroups WhereCore's flat `Sources/` into concern-based subdirectories. One commit per step.

**Guiding principle:** an AGENTS.md keeps invariants, lessons, conventions, and recipes; it drops module lists, directory trees, and dependency catalogs that mirror `Package.swift` / `Project.swift`. The root file's libraries list was already stale (missing `WhereIntents` and `SwiftDataInspector`) — proof that content rots.

## Commits

1. **GitHub rules** — new root **GitHub** section: use `gh` for all GitHub interaction; open PRs ready-for-review, never draft; keep open PRs current by pushing each local commit. "Working on PR feedback" now requires replying to a comment when a commit resolves it, and filing deferred feedback durably instead of dropping it.
2. **Root trim** — libraries/targets enumerations become pointers at the manifests; directory layout collapses to the module skeleton. The add-a-target recipe, CI-scheme rule, and the double-linking lesson stay verbatim.
3. **Where trim** — Modules tree becomes a one-paragraph layering stack; developer-overlay tour and feature narration cut; the stylesheet paragraph now points at `WhereUI/AGENTS.md`. 222 → 189 lines.
4. **Module trims** — WhereUI / WhereCore / WhereIntents lose their `Package.swift`-mirroring dependency listings; PeriscopeCore loses its directory catalog. All invariants and the full `WhereStylesheet` guidance stay.
5. **WhereCore regroup** — pure `git mv`, no behavior change: stragglers fold into `Backup/`, `Reminders/`, `Widgets/`, `Location/`, `Evidence/`, `DataResolution/`, `Persistence/`; new `Days/`, `Reporting/`, `Journal/`, `Preferences/` groups. Only `WhereServices*` and `WhereLog` remain top-level. Tests stay flat, matching PeriscopeCore.

## New lessons codified (mined from the last 30 days of PR review)

- Prefer synthesized `Codable`; persisted wire formats must survive Swift renames; hand-written conformances document their load-bearing reason (#85, #86)
- Retain block-based notification-observer tokens; every `start` has a paired `stop()` (#69)
- Group large flat types into sub-structs / child types per behavioral area (#47, #69)
- Test doubles conform to the production protocol, never an enum switch in a production type (#54)
- Custom full-screen surfaces are accessibility-modal and post focus notifications (#70)
- Derive UI dimensions from the live UI (preference keys, `@ScaledMetric`, semantic font styles) instead of repeating magic numbers (#38, #70, #83)
- WhereCore: post-write reconciliation is defined once — writes/imports route through `DayJournal.reconcileAfterDayChange()`; cross-collaborator hooks are a single closure (#80, #88)

## Verification

- `./swiftformat --lint` clean; `./sync-agents` run after each docs commit
- Full `Stuff-iOS-Tests` scheme after the regroup: 1183 tests, 0 failures

Made with [Cursor](https://cursor.com)
kyleve added a commit that referenced this pull request Jul 15, 2026
…store-backed tracked regions (#87)

## What & why

Prepares RegionKit for "more locations" ahead of letting users pick their own regions. Two coupled shifts:

1. **Split** the ~2.5 MB monolithic `us-states.geojson` into one GeoJSON file per region, loaded **on demand** — so we only ever parse the regions we track, not the whole US at launch.
2. **Replace** the hardcoded five-case `Region` enum with a **data-driven catalog** (a bundled `regions.json` manifest + a `Region` value type), so any US state (50 + DC + PR) plus Canada/EU are available regions. The user's **tracked** regions live in the synced SwiftData store (WhereCore owns "which"; RegionKit stays the agnostic geometry/lookup engine).

Onboarding's region picker and user-chosen per-region styling are **out of scope** here (designed for, not built).

## What changed

- **RegionKit — data-driven catalog + on-demand geometry**
  - `Region` is now a `Hashable`/`Codable` value type over a stable id (`us-CA`, `canada`, …) with a well-known `.other` and conveniences (`.california`, etc.). `RegionCatalog` loads the bundled `regions.json` manifest (all/name/localizedName/geometry file, canonical order). Adding a region is now **pure data** — a manifest + geojson change generated by `Tools/generate-regions.rb`, no code.
  - `RegionAttributor(for:)` loads only the passed regions' files; `.all` covers the whole catalog, `.shared` the default four. `RegionAttributing` abstracts the engine so the app can supply a live, swappable attributor. `RegionGeometryCatalog.outlines(for:attributor:)` takes an explicit attributor.
- **WhereCore — store-backed, synced tracked regions + live attributor**
  - Tracked regions persist as **one `SDTrackedRegion` row per region** (so concurrent cross-device edits merge instead of last-write-wins), read as a `Set` defaulting to the four until the user chooses.
  - `RegionAttribution` rebuilds the attributor from the tracked set on the store's `changes()` signal (local edit or remote CloudKit import). `WhereServices.make(...)` (async) derives it from the store; the app launch and the App Intents process (`WhereServices.forIntents()`, now async) both attribute against the same synced set. `make(...)` is the **sole public assembly entry**; the synchronous `init` is `@_spi(Testing)` for tests/previews.
- **WhereUI / WhereIntents — catalog-driven**
  - `RegionStyle` is id-keyed with an isolated, disposable bespoke-override table (the seam user-chosen styling will replace). Ordering uses `Region.inCanonicalOrder(_:)` instead of scanning `Region.allCases`.
  - Siri's "pick a region" suggestions and the Spotlight index surface the **tracked** set; `RegionEntity` id resolution stays **full-catalog** (so "days in Texas" answers even when untracked).
- **Backup** — tracked regions round-trip in the archive (additive field; v1 manifests decode to `[]`); `.replace` restores the set exactly, `.merge` unions it.

## Key decisions / tradeoffs

- **Clean id scheme** (`us-CA` vs `california`): existing persisted region raw values change; data is re-exported/re-imported. The `NAME → id` map lives in `Tools/generate-regions.rb`.
- **Localization tradeoff:** region names come from the manifest (with an optional `localizationKey` override), so region names lose static string-catalog extraction — consistent with how the App Intents `RegionEntity` already resolves names at runtime.
- **Attribution priority** is the manifest's canonical order (regions are mutually exclusive at our resolution).

## Deferred (follow-ups)

- The onboarding region picker and user-chosen per-region color/emoji/symbol/tint.
- Seeding semantics (the default four apply only at zero rows; the picker will seed/replace) and materializing the implicit default before the first explicit edit.
- **Untracking a region:** the store currently *deletes* the tracked-region row, which would re-attribute that region's past GPS days to `.other` on re-aggregation (manual days, stored as region sets, are safe). A `TODO` defers the soft-delete (mark inactive + load every ever-tracked region so history stays stable) to the picker work that defines the untrack UX. Not user-reachable today — nothing calls `setTrackedRegion(false)` outside tests.

## Testing

- Full `Stuff-iOS-Tests` scheme green; `./swiftformat --lint` clean.
- New/reworked coverage: catalog + value-type (`RegionTests`), subset-only-loads (`RegionAttributorTests`), rebuild-on-`changes()` + store round-trip (`RegionAttributionTests`, `TrackedRegionStoreTests`), end-to-end `make()` wiring (`WhereServicesTests`), tracked-set entity/suggestions (`RegionEntityTests`), and backup round-trip / legacy-manifest decode (`BackupServiceTests`, `BackupCoordinatorTests`).

## Review pass

- **Self-review:** log (not swallow) tracked-region read failures; deterministic (canonical-ordered) attributor build; surface unknown stored region ids; backup round-trip (above).
- **PR review feedback:** consolidated the RegionKit bundled-data docs into the main `README.md` (removed the separate `Resources/README.md`); documented why `Region`'s `Codable` is hand-written (bare id string vs the synthesized keyed object); made the sync `WhereServices.init` `@_spi(Testing)` so `make()` is the sole public entry; and TODO'd the untrack soft-delete (see Deferred).

## Kept current with `main`

Merged `#85` (CalendarDay), `#86` (Periscope JournalKit), `#88` (backup `onImport` reconcile hook), and `#89` (streamlined AGENTS + WhereCore source regroup). Conflicts resolved: the backup `formatVersion` (kept v2 **and** the additive `trackedRegions` field); the `BackupCoordinatorTests` harness swap (kept both the tracked-region tests and `#88`'s `onImport` hook test); and `Where/AGENTS.md` (took `#89`'s streamlined Modules section, but kept the accurate manifest-backed RegionKit localization note over `#89`'s stale "static keys" wording). My changes rode the source regroup into `Backup/`, `Persistence/`, `Days/`, `DataResolution/`, etc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant